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''ABSTRACT -i • 

; • . The dimensions of educational evaluation Include 
conceptualization and technology of evaluation. The" methods have been 
drawn from the. area of educational and psychological research. The 
techniques of evaluation need to' serve* the inf ormation needs, of 

Clients; address the central .value issues, deal with situational 
realities, meet requirement* of probity, and satisfy needs for 

€ver»eity. Many researchers have developed new methodology. There is ja 
\i«ed to elert evaluators to the availability Of new techniques. The? 
should be encouraged to try out and report the results of new 
techniques,- as well as developing new methods. The methodology must 

TTifTthe 1ieeds~of education. The application of evaluation was 
identified as_a problem area -because there is often insufficient 
staff and resources to implement systematic evaluation. There is an 
inability to prevent, corruption by political forces. Most of an 
institution's evaluation efforts tend to be committed to satisfying 
external requirements for ^valuation -while ignoring infernally based 
needs. Professional support for evaluators has strengthened and there 
has been substantial progress in the professionalisation of. 

. educational evaluators. The analysis of educational evaluation 

-illustrates the-need to improve research, training, and financial 

i support for this profession. (DWH) 
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*- ' • 6 / Introduction ' - . 

.. " . • • .. \ ■ •• 

- the increased investments in education a^a means* of solving social 
problems, during the 1960's, led society to require educators to. evaluate 
the'ir work. In the course of responding to this outside pressure, educators 
expanded their evaluation efforts not only to meet society's accountability 
demands, but als"o to provide direction for improving their various projects 
£nd programs, ',4s a consequence of this increased attention to educational 
. evaluatiojvthousands of persons became involved in providing specialized - 
• evaluation service; and. many of them found that they were ill prepared to 

meet .th e . cii&UangMfiuft ,_1 9,69) . 

In attempting tcfideritify and meet their common needs, these persons 
inevitably began to forge a. profession of educational evaluation. This 
profession is visible in-many forms thatwere not present -just 10 years 

.ago-. The field's recently acquired characteristics /include its -journals* 
newsletters^ and books; its organizations and conventions';" its companies 
and institutional .units; its training .and research programs; and its 
standards. While the profession in many respects is immatur a, there can be 

-no doubt that increasingly it has become an identifiable, component of the 
broader governmental .and professional^ establishment of education. • The pre- 
dicti,on, commonly heard in the mid 1960's 1 , that. formalized educational 
evaluation -was a fad anji would soon disappear, proved false, and there, are 

' Wong indications that this field will continue €o grow in importance and 

F * . ,. '. ' • . ' • *• 

. sophistication^ . . 

r » . . . ■» , . . 

In order' to foster positive growth of their field, educational 
evaluators should, identify and attend as carefully. as they can to their ' 
past successes and faiTures. f Otherwise, they are doomed .to repeat their 
. mistakes and—equally debilitating— not to sustain and build on their 



successes. Ongoing review and. improvement of one's service is clearly the 
hallmark of a professional, and", by .extension, of a profession- An important / 
fynctiorl of organizations, such ^as -the Evaluation Network, 1s, to foster such 
review and to provide direction for -needed improvements. 

As a modest contribution to helping EN t^ serve this function, I am 
pleased to' respond to Bob Ingle's invitation to examine the history and* 
future directions of educational evaluation. Largely, I -will Confine my 
review to the last .fifteen 'years and will consider four dimensions that I ~\ 
believe are important in the examination of any profession. I will also offer 
a few recommendations directed: at overcoming problems and building on past 

, j . « 

achi evements . o •; • ; .* m j ' 

• - ' . J * . • , ' * 

I " 1. " Conceptualization of Evaluation 

* ■ • ~ 'I 

Clearly, one of the problems most evident in the literature of evaluation 1 

is the confusion and controversy over the term evaluation. A particular ^ 

'problem, in this regard, is that pioneers in evaluation—and especially fheir 

. deciples"--tended to equate evaluation to the use of some preferred technique-. 

They linked the term tp behavioral .objectives (Tyler, 1949), standardized 

tests (Ebel, 1965), experimental designs (Campbell and' Stanley, 1963), and expert 5 

reviews (National Study of Secondary School* Evaluation, 1969); While"these tech- 

- . . o • 

niques : undoubtedly are useful in evaluation, equating any one of t(iem to evaluation 
is a serious mistake (Fishes 1951). The consequences of using such equations 
are narrow assessments, feedback limited to post hoc. results, impractical/ 
inflexible procedural plans, and/or threat brought~on by vesting all authority 
for judgment cf a program in a party having!. little or no responsibility for 0 

acting '•on the recommendations'. • • '-■> 

■ ? ' \ * 

Importantr additions tQ;the suggestions for defining evaluation inclgde 



Cronbach'-s emphas?Sv(1963) on evaluation as* a process for guiding improvement ^ 
efforts, the recommendation by the POK 'Study Committee on Evaluation 
( Stuff! ebeam, et al\, 4971) that evaluation s;hou1d be viewed. as a process for^ • 
identifying and judging, decision alternatives, Stag's- ( 1 967") equating of ^ 
'evaluation to a process of description and judgment, Scriven's (1967) 'emphasis 

' Of. 

on a systematic approach to assessment. of worth and, merit, and Eisner's (1975) 
characterization of evaluation as conn6iseurs4iip and criticism. These and * 
other authors have extended and explored/the meaning of evaluation, beyond 
mererdefinitions, by offering extensive theoretical formulations ^(Stufflebeam: 
and Webster, 1980). And., Very recently,, Cronbach and Associates (-1980)' , " *£. 
provided an in-depth view of the meaning of^aluation through their end^chtment 
of .current. theory and jpractice, their cal'1 fofta reformation', their'-positing of 
95 theses, and; their -outlining of needed .improvements. The Cironbaeh review '.- • 

' revealed that the disagreement, prevalent in the late 1960's about whether . . V 
evaluation should be mainly formatiye or summative, i-s .still present. 

. A very general view, in. the, form of 30.' articulated and illustrated 
standards, wa"s provided by the J.oint Committee dh Stahda>ds for Education^ 

• Evaluation (1981). Their formulation v which is reflective of a^ common 
dictionary' definition, is that evaluation is^the systematic. Assessment of the 
worth or merit of some, object. While this formulation 'lis consistent with* 
"Scriven's position" that evaluation necessarily involves value determinatio ns v 
it acknowledges (particularly 'through. its Valuation?! Interpretation -Standard) 

.the potential utility, under different sets qf circumstances, of several 

•different ways of assigning value meaning to findings. These options include' 

matching outcomes to valued objectives,: collecting and analyzing judgments 

'offered by a wide range of inteVested parties, assessing the extent that 

* ** ' ' . ' * 

attainments are responsive 'to assessed needs, charging the evalaiator to offer 



i . - " 

an overall assessment of worth or merit, and advising the'audience for,the 
* «► . 

evaluation to arrive, at a judgment. The fact that the Committee dicPnot 

' * • * * 

recoLiend any .of these over the! others .is re/lective'of the state of the.art 

» • * » ** 

devaluation-,' since respected leaders have proposed the different approaches 

' • ' * " ; ! ■ . ' . ■ o 

to valuing, while research has not demonstrated the superiority of any. of / 

them. , f 

- -The preceding discussion is indicative of both the substantial progress^* 

of educational evaluated in conceptualizing evaluation and 'the^eed-for - 

further work. Clearly, some members of the educational' evaluation community 

•have adyanceji in their flexibility and careful thinking about, Evaluation. 

On the other .hand, many practitioners ;of educational evaluation have persisted ■ 

to \ise some narrow cdncept of ^evaluation without developing a defensible « 4 

raftidi^e forjts usg, 'Also, there has been very little empirical research on 

the relative merits pf different approaches to evaluation. "Clearly, the field 

of evaluation could 'profit from systematic examinations of < the feasibility, 

costs, and benefits of competing conceptualizations, and from more effective 

dducation—of^the fini^range^of participants in the educational evaluation 

enterprise— about what Stake' (1967) has termed the "fufi courftenance of evaluation 

- ». - 

.'. • • -• . . i I- - .'' 

■ * ' / • . •. 2v . Technology of Evaluation . 
Qlosely linked to the problems "of 'conceptualizing evaluati on are tbps e 



linked to the technology t pf evaluation. The methods used most often in 
educational ^valuation have been drawn from thd afea of educational and 

psychological research. Especially these inclutfe^comparati've experiments * 

* * * * 

as a means of assigning students to programs, standardized tests for 

** , * * * 

obtaining outcome data, and $ analysis of variance for examining and inter- 
preting the obtained results. A pervasive -problem is that these techniques^ 



are poorly fitted* to the characteristics' and special needs of the- vast 
majority of evaluation studies (Wolf, 1981). They require assumptions" that 

* c * 

often could not be met, and, even if they cool d^ these techniques typically 

address questions other than. 'those of primary concern*" to the audiences for 

•*•»'' • 

evaluati'ons-.^oreover, the use. of • these techniques without regard for . 

; ) ' , ' ' • * *. 

satisfyingijheir required assumptions has led, to the justifiable charge that 

evaluations freoroently are no more than poor research studies. A costly con- 

sequence- of this problem, has beetvan enormous -overinvestment by some of the 

I field's most talented researchers in attempts to engineer research methods 

to meet the requirements of evaluation.. The payoffs fronr these efforts 

have been miniscule compared to the costs of development and field testing. 

flnly recently have educational- evaluators. begun- to realize that evaluation 

' • . * 

needs a respectable methodology that, is built' from the ground up '. That iu, 

* " • ■ ■ . ** 

the 'techniques of .evaluation must be built t6 serve the information needs of 

the clients of evaluation,- to. addresj the. central value issues, to deal with 

situational realities, to meet* the requirements" of probity,- and to satisfy 

needs' for veracity, while the field is far from developing a fujly functional 

methodology that meets' these requirements' there have been some promising 

developments. These include Goal Free Evaluation (Scriven,, 1974, Evers, 1980), 

f \ v • ■ * 

Adversary-Advocacy teams (Stake and Gerde, 1974), Advocate-Teams ( Rgi nhard , 

-nW)T11etan^KislGlass,'l976, Krol, 1978), Responsive Evaluation. (Stake, 

._._.*» 

1 9 7.5)^ and Natural is tic Evaluation - (Guba and Lincoln 1981). Under the leader-.- 
a ship of Nick^Smith (1978) a panel'of writers has examined the" applicability 

to evaluation of a wide range of investigatory techniques drawn fronfa . 
^variety of fields^ Eisner (1975)'and his students have explored and 

developed techniques for appTyihg~~tbe Connoiseurship model . Webster 

(1975) and his colleagues have operational i zed the CIPP model . t 



Stake (1978) has adapted case study methods for use in evaluation. Roth frW7fc 
Suarez {>9'6Q), Scriven (1977), and others ha've begun to make both conceptual and 
operational, sense of the crucial yet 'illusive. concept of needs assessment."" - ;~- 
Personnel of the Toledo-Public Schools' have collaborated wi^ Bunda (1980) and 
Ridings (T980). to' devise catalogs of. evaluative criteria and associated 
instruments' as a means of helping teachers and administrators to gear thei r "data 
collection effdrts-to their information requirements. 
^ ' Finally, a great deal of work has been done to encorporate the use of 
Objeoti-ves referenced tests in evaluation "studies'. A particularly fruitful 
application !of this latter technique^ is seen in curriculum embedded evaluations 
which provide teachers and student'^ with an ongoing assessment of attainments in 
relation to the sequential objectives of a fcurHculum. Important v(prk in this . 

area is being carried out in the Cfijcago 'Schools and *by the Curriculum and 

* ■ 

Instruction Department in the Dade County Schools (Chase 1980). ^ 

• © - - - ~ " * ~ , 

The efforts* to expand and "improve the methodology of evaluation have had 

> ' * * 

some benefitical effects.- There has been more •acceptance of'\he neecj for 
alternatives .to classical research methods-/, The proponents ^n^njtjj:s_ r of— % - 
experimental design, as an optimal means o^F "evaluation, haye'become less^ 
polarized and, less emotional' in their exchange about ^what/methods are needed . • 
and w hat will w ork i n erimia flnn n l ov n lunti o n setting s-tSferk, tsre-l). ■ • " 
•In spiie of a growing search for appropriate methods, increased communica.- 
tion and understanding, among the leading methodolog.ists, and; the development of 
new techniques, the actual practice of evaluation has* changed, very little in the 
great majority of settings. Clearly, there is a need for expanded efforts to 
educate .evaluatocs to, the availability of nev/ techniques, to try out and 
report* the results of using the new techniques, and to develop additibfal 
techniques- In *11 of these efforts* the emphasis must be on making the 
methodology fit the needs of education, rather th'an vice versa (Kaplan,, 1964). 



V A\ third- problem area involves the implementation *.of evaluation., 



ForemOiSt io^is ar,$a is insufficient staff- and resources in many educatfonal. . 
\n$MtuWn^to\*arry' out systematic evaluation. Another is the inability to 
jvent evaluations from being corrupted by political forces. . -Finally, there 
* is. a widespread tendency to commit most or all of an. -institution's evaluation 
efforts- to satisfying, external requirements for^Valuation^ while leaving 1 

^interhiir^ • ' "*! 

■«•"■«' ■ "" - . -' " ' ' ' "'■ ' > ' " " 

The consequences of these deficiencies are readily apparent in many . 

educational institutions, and are" indicative of a need for concerted effdrts 

to increase and improve the application of evaluation. Systematic evaluation 

. * ' ■ ■ \ ■■■ 

of education is generally -present only in the large urban school districts 
that have sufficient resources to. Maintain a staff of evaluation specialists;. 

and even in these districts evaluation* services are heavily concentrated on . 

" . . ■ • • .- » . 

issuing accountability reports for special projects. funded from the outside 

■ ^d. deaVmainly^i|IPdistrict-level concerns to the exclusion of school .awL - 

* - i t % 1 * 

■ classroom concerns.. Compared to the Jleve.l of evaluation .activity in urban 
school distHctsl"''systematicprogr^m_jYA ir * , 

"^mTrr^chooTTdistricts and 1 in colleges and universities. Also there have 

•• I •'• ■ > * 

been Repeated reports (Brickell \ 1976, House 1973 Sroufe 1977) about the 

vulnerability, of evaluations to political sabatage; ' . . • 

Despite the generally negative report on institutional capabilities to 

conduct evaluation', there haye been some important developments. Some districts 

have set up and-operated excel/lent offices of'eWluatioh (especially Saginaw, . 
.Michigan.; Dallas, Texas; Atlanta, Georgia; Portland, Oregon; Los Angeles » ■ 

County; Cfncinatti, Ohio; Lansing, Michigan; Toledo, Ohio; Austin, Texas; 

and Philadelphia, Pennsylvania). Also, a number of corporations-- such as Abfc ' 
—Associates,- System Development .Corporation, and American Institutes for 



•Research— have- developed and delivered-rtigh quality evaluation services *4> . ' 

educational institutions: . Recently, Cronbach and Associates (1980) suggested 

the usi! of "social problems study 'groups" funded over a Tong period .of time to 

sustain ongoing evaluation .of high priority national concerns i*n education and 

other areas, This arrangement would have the advantage of concentrating 
• « « 

resources, in Iqng-term programmatic investigations aimed at the formulation 

and implementation of sound policies. ' . 

There have also been some noteable developments to deal with \political 

realities. The Saginaw Schools (Adams -1970) developed. Board of ..Education , 

Policies to'guide and govern evaluation in the District. Tfie Dallas System , 

set up a Board of Education Committee on Evaluation and used it to help 

* '* I , ' * 

focus district evaluation efforts on Board informationji^e^^Tand to increase 

\ m - — . A 

the Board members' use of evaluation. While such Board Committees have been 

controversial (*D. Magazine, 1980), the Dallas experience' in* this realm 

merits careful study. • • 

One of the most promising developments towards increasing the use of 

evaluation -a t-thT school level' is seen in Atlanta's assignment of dual roles 

to evaluators; each evaluator serves a certain technical function such as 

statistical analysis and a general liaison function to a sample of schools. 

In "addition, each school maintains a curriculum improvement committee' which 

includes the assigned district level ^valuator and is.Qharged to conduct; t 

evaluation to identify and address curriculum improvement needs in the school. 

This arrangement provides for a strbng central evaluation functfon and 

■ /. « 

guidance and, technical backup for school level evaluation. 

The problem observing both school and 'district evaluation functions is 

peculiar to American Public Education. In other countries public^ scH ols 

report directly to a central governmental "authority and .are not grouped intg 



districts. Some of the most advanced work J n school level evaluation . ^ 

' is to be seerrin the private schools of, the United States aod vr the private. 

and government' supported schools in Victpnla, Australia, 

* \ A final ; point worth noting, is that several studies have been done or are 

* ' " * . ,, 

ln r progress to illuminate aod .assess the strengths and weaknesses of evaluation 

capabilities' and practices in school districts' (Chase, 1 9 80 Kennedy 1980, 

Alkin 1979, .and Ring 1981). These are an important resource^for taking stock of 

•what has worked* what hasn't* and what yet needs to be done to increase the capa- 

• ^ „ V . 

/ • * 

bilities of educational institutions to carry out vftal> evaluation functions. 

* 

Furrier studies of evaluation practice at the school level aimed at assessing the 
state o'f the art, in general , and' promising evaluation practices-, in particular, 
could provide-principals and teachers with' valuable Information by which to 
improve their use of evaluation. \- 

4. ' Professional Support for Evaluation 

As th6 field of educational evaluation has grown-, it has experienced c 
needs 'that are dommon to emerging professions. While the° field has 
advanced considerably" in dealing with these needl, it- is instructive to 
consider the great deficiencies in this area that existed just 15 year^ ago. 

At that time, practitioners of educational evaluation faced an identify ^ 
crisis. They weren't sure whether they should try to*be v researchers, * 
k testers, administrators, teachers, or philosophers/ They were unclear about 
what special qualifications they should possess. There was no professional 
organization that concentrated on their particular problems. There were no 
specialized journals through which they could exchange information about 
their work. There was essentially no literature about educational evaluation. 
Theresas a paucity ^of preservtce and inservice training' opportunities in 
evaluation. And there were no .articulated standards of good practice, except 
for those confined to educational* and- psychological tests'. . 



The effects of these deficiencies were apparent' to any one. who looked' 
'closely at 1 educational evaluation during the late 1960's. The field was j 
amorphous and fragmented. Many evaluations were carried out by .untrained 

personnel others were. performed, by research method ologists wno tried t o ^_ 

force' educational -Situations .to fit. the'ir, methods (Guba 1966). Evaluation 
. ^studies were characterized by great confusion, anxiety, and animosity. In 
•* general, the budding field of educational' evaluation had li ttle. stature 
<~ and poetical clout. f ' 

_ Agatnst thi.s backdrop, the progress of educational evaluators m 

«"-*•" 

professionalizing tljeir field during the 1970's, appears to be substantial. . 

Educational- Evaluation and Policy Analysis , S tudies in Evaluation , CEDR 

4 Quarterly , and Evaluation .News have proved to be.excejlent media for. 

« . . ^ 

recording and disseminating information about progress in educational- 

• '• • ',«••<•' 

evaluation. Books and monographs by Po'pham (1974), Provus (T£71), 

^Bloom, Hastings, and Madaus (1971), .Scriven (1980), Worthen and , m 
Sanders (1973 >, Glass (1976), and many others have provided an enormous' 
amount Of information about the educational evaluation enterprise.- The, May 
• 12th group; Division H of AERA, the Evaluation' Network, and the Evaluation, ># 
Research. Society of America, have afforded excellent opportunities' for profes- 
sional exchange among persons concerned with, the evaluation- of education- and 
other social programs.. Many universities have begun to offer at least one . . . 
course in evaluation methodology (as distinct from research methodology),- 
and a few— such as the University of Illinois, Stanford University, Boston 

" College, UCLA, the Lniversity of Minnesota, and Western Michigan University- 
have 'developed graduate programs'.™ evaluation. For seven years the U.S. Offee 

of Education has sponsored a national program of ihservice training in evaluati 
of special education (Brinkerhoff , in press), and several professional 



on 
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j » 

organizations have offered workshops ant}, institutes on various evaluation 
topics. A/ewcenters have been establtshed and maintained to.conu&ct research 
ahd development on evaluation; these include the evaluation- unit of the North- 1 
west Regional Educational Laboratory, the Center for the" Study of Evaluation 
at UCLA, the Stafford Evaluation Consortium, the Center for Instructional 
Research and Curriculum Evaluation at the University of 111 inois,\and the 
Evaluation Center at Western Michigan University". The state of-Loiiisiana 
has established a policy and program fo^certifying evaluators (Peck 1.981), 
and Dick Johnson (1980), has issued a first draft of a directory of evaluators 
and evaluation agencies. Increasingly, the field has looked to meta evaluation 
(Scriven 1^75, Stufflebeam 1978) as a means of assuring and checking 'the quality 
of evaluations* Finally. 'a joint committee (Joint Committee, 1981, a.), 

appointed by 12 professional organizations has issue4 ; a comprehensive set of 
standards for judging evaluations of educational programs, prpjects, and 0 - • 
materials, and has established a mechanism (Joint Committee, 198i, b.) by which 
to review and, revise the Standards* and assist the field to use them. In addition 
several other sets of standards with relevance for educational evaluation. (see 
the May, 1981 issue of ^ Evaluation News ) rave been issued 

This substantial'professional development in educational evaluation has 
produced mixed results- While there is undoubtedly better and more communica- 
tion in the field, there las also been an enormous amount of "chatter" (Cronbach, 
1981).' While progress has been made through better and more training and 
certification efforts to ensure that institutions can obtain services from 
qualified evaluators, there §r% worries (Stake, 1981) that this movement may 
result in a club that narrows the practice of evaluation and excludes persons 

'from joining, it for no good rieasqn. The cooperation among professional 

t 

organizations concerned with educational evaluation, fostered by the Joint 



Committee on Standards for Educational Evaluation, is a promising but fragile 
arrangement for promoting the conduct and use of high quality evaluation work. 
And the creation of new professional organizations has increased communication 
and reduced some of the fragmentation in the evaluation field; but fairly 
sharp divisions between Division H, the Evaluation Network, and the Evaluation 
Research Society continue to exist. ., 

in order to sustain. gains and work on solving some of the persistent 
problems in the professional support- area,. several jrioves are in order. 
Editorial boards of evaluation journals should review and tighten their 
criteria for choosing articles and, through more careful refereeing, should 
attempt to increase the proportion of high quality published material . Annual 
"evaluation circuses" in which evaluators of education and other social program 
areas meet together without, giving up their primary evaluation organization 
should be continued; and Division. H of AERA should be encouraged to participate 
along with EN and ERS. The organizations and persons with professional 

* « 

interests in and allegiance to the field, of educational evaluation should, 
sustain the work of the Joint Committee and use it to promote better conduct 
'and use of evaluation and improved credibility for the profession. Concerted 
efforts should be made to encoVporate evaluation training into the professional . 
preparation programs of educators, 'especially educational administrators. .The' 
majbr evaluation organizations must develop and sell a strong case for financial 
support of the educational evaluation enterprise. Considering all they spend on 
evaluation, government agencies should continue to invest in efforts to improve 
the evaluation enterprise, including research, development, training,^and pro- 
fessional support.. Hopefully, private foundations will see the wisdom of 

> 

helping to fund the improvement of evaluation, since sound evaluation is 

> • * „ v 

\ essential for improving education. 
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Closing 

t 

The preceding review portrays educational evaluation as a dynamic, yet - 
immature- profession. The-g^ins in this field over the past 15 years are 
impressive i but there areWny obvious deficiencies and insufficient 
evidence about impacts on teaching and learning. Strengths and weaknesses 
-have -been outlined along with suggestions for improvement in four areas: 1 
the conceptualization of evaluation, techniques, applications, and professional 
-support. The. pervasive theme in this analysis points to needs to improve 
research; training, and financial support 'for educational evaluation,. However, 
leaders of the educational evaluation profession must ensure that efforts 
to improve thjs profession afe geared not to servejthe private and corporate 
needs of evaluator's, but the evaluate, service needs of educators and their 
clients. Ultimately the value of educational evaluation must be judged, in 
terms of its contributions to improving learning, teaching, and educational 
administration. All qf us in the educational evaluation .business would do 
well to remember and uJe this basic principle to quihe and examine our work. 
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